Relevance Judgments
نویسنده
چکیده
Precision and recall are used to evaluate information retrieval systems. These in turn are based on identifying documents relevant to a query in a document collection. However, the notion of relevance is highly subjective and thus it can be debated whether measurements based on relevance are reliable. In this paper studies investigating this issue are surveyed and the results of those studies are presented.
منابع مشابه
Crowdsourcing Relevance Assessments: The Unexpected Benefits of Limiting the Time to Judge
Crowdsourcing has become an alternative approach to collect relevance judgments at scale thanks to the availability of crowdsourcing platforms and quality control techniques that allow to obtain reliable results. Previous work has used crowdsourcing to ask multiple crowd workers to judge the relevance of a document with respect to a query and studied how to best aggregate multiple judgments of ...
متن کاملPhilosophy of IR Evaluation
• System evaluation: how good are document rankings? • User-based evaluation: how satisfied is user? NIST Why do system evaluation? • Allows sufficient control of variables to increase power of comparative experiments – laboratory tests less expensive – laboratory tests more diagnostic – laboratory tests necessarily an abstraction • It works! – numerous examples of techniques developed in the l...
متن کاملPolicy Capturing Models for Multi-faceted Relevance Judgments
We applied policy capturing and bootstrapping methods to investigate the relevance judgment process, with a particular focus on understanding how judges summarize an overall relevance judgment from five specific aspects of relevance. Our data come from relevance judgments made in the development of the MALACH (Multilingual Access to Large Spoken ArCHives) Speech Retrieval Test Collection. We de...
متن کاملManaging the Quality of Large-Scale Crowdsourcing
Crowdsourcing can be used to obtain relevance judgments needed for the evaluation of information retrieval systems. However, the quality of crowdsourced relevance judgments may be questionable; a substantial amount of workers appear to spam HITs in order to maximize their hourly wages, and workers may know less than expert annotators about the topic being queried. The task for the TREC 2011 Cro...
متن کاملUnifying morality's influence on non-moral judgments: The relevance of alternative possibilities.
Past work has demonstrated that people's moral judgments can influence their judgments in a number of domains that might seem to involve straightforward matters of fact, including judgments about freedom, causation, the doing/allowing distinction, and intentional action. The present studies explore whether the effect of morality in these four domains can be explained by changes in the relevance...
متن کاملReducing Reliance on Relevance Judgments for System Comparison by Using Expectation-Maximization
Relevance judgments are often the most expensive part of information retrieval evaluation, and techniques for comparing retrieval systems using fewer relevance judgments have received significant attention in recent years. This paper proposes a novel system comparison method using an expectationmaximization algorithm. In the expectation step, real-valued pseudo-judgments are estimated from a se...
متن کامل